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BACKGROUND OF THE INVENTION 
1. Field of the Invention 

The present invention relates to an image processing 

15 method that may be adopted to perform low-pass filtering 
processing on image data obtained by performing an image 
capturing operation at an image capturing element in a 
digital still camera or the like. The present invention 
also relates to a recording medium that stores an image 

20 processing program utilized to implement the image 

processing method on a computer and a data signal used to 
transmit the image processing program via a transmission 
line. Furthermore, the present invention relates to an 
image capturing apparatus and an electronic camera that 

25 executes the image processing method. 
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2 . Description of the Related Art 

In the known art, noise in image data is usually 
removed by performing low-pass filtering processing of the 
image data. However, while a noise removal effect is 
5 achieved through normal isotopic filtering processing, 
i.e., through filtering processing performed uniformly 
over the entire image plane, the original high-definition 
structure of the image data becomes lost . 



10 SUMMARY OF THE INVENTION 

An object of the present invention is to provide an 
image processing technology that enables low-pass 
filtering processing to be performed on image data without 
losing the fine structure of the image. 

15 Another object of the present invention is to provide 

an image processing technology that makes it possible to 
remove jaggies (steps occurring at inclined edges or the 
like) which have been difficult to remove in the prior art. 
The present invention is adopted in an image 

20 processing method, through which low-pass filtering is 

performed on image data, comprising a similarity judging 
step in which degrees of similarity among pixels are 
judged along at least four directions in a local area 
containing a target pixel, i.e., a pixel undergoing low- 

25 pass filtering processing and a direction-dependent low- 
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pass filtering step in which a pixel value for the target 
pixel is obtained by performing weighted averaging of the 
pixel values of pixels around the target pixel by 
increasing the weighting rates along directions 
5 manifesting a marked similarity based upon the results the 
judgement made in the similarity judging step and adding 
them to the pixel value of the target pixel, to achieve 
the objects described above. 

In the image processing method described above, pixel 

10 lines are set along at least four directions (e.g., 

"vertical", "horizontal", "upward diagonal" and "downward 
diagonal") in the local area containing the target pixel. 
Differences in characteristics among the pixels (level 
difference, hue difference, saturation difference and the 

15 like) for the individual pixel lines are detected to judge 
the pixel similarity values along the individual 
directions. Next, a weighted averaging operation is 
performed. In the weighted averaging operation, weighted 
pixel values of the pixels around the target pixel are 

20 added to the pixel value of the target pixel and the 

result of addition is divided by the sum of the weights, a 
weighting rate along a direction manifesting a marked 
pixel similarity becoming increased. 

As a result, since weighted averaging is performed at 

25 a low weighting rate (including zero) along directions 
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manifesting only a small degree of pixel similarity, the 
image structure of the target pixel is not compromised to 
a significant degree. In addition, since weighted 
averaging is performed at a high weighting rate along 
5 directions manifesting a high degree of pixel similarity, 
slight structure (such as noise) at the target pixel can 
be effectively suppressed. 

It is desirable to judge similarity by using either 
one or both of the characteristics differences in (1) and 
10 (2) below in the similarity judging step. 

(1) Characteristics differences among a plurality of 
pixels located on lines passing through the target pixel 
along specific directions. 

(2) Characteristics differences among a plurality of 
15 pixels located on lines passing near the target pixel 

along specific directions. 

(1) and (2) are explained individually below. 
© Similarity judgment implemented by using only the 
characteristics differences in (1) above. 

20 It is judged that the similarity between the target 

pixel and nearby pixels (hereafter referred to as 
"similarity on a central line") is high if the 
characteristics differences in (1) are small. 

When the weighting rates are determined based upon 

25 the degrees of similarity on the central lines in this 
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manner, slight structure of the target pixel (i.e., noise) 
can be rigorously suppressed while sustaining the image 
structure (such as the edge structure) of the target pixel 
in a reliable manner. 
5 © Similarity judgment implemented by using only 
characteristics differences in (2) above 

It is judged that the degree of similarity between 
nearby pixels (hereafter referred to as "similarity on an 
adjacent line") is high on a line adjacent, to the target 

10 pixel if the characteristics differences in (2) above are 
small. When the weighting rates are determined based 
upon the degrees of similarity on such adjacent lines, 
j aggies can be removed effectively, as shown in FIGS. 3A 
- 3E, to be explained in reference to an embodiment later. 

15 The following is an explanation of how such advantages are 
achieved. 

Normally, noticeable j aggies occur on an inclined 
edge or on the boundaries of an inclined line. The 
continuity is markedly interrupted at a pixel manifesting 
20 such a step in relation to nearby pixels. Thus, the 

characteristics differences among the pixel manifesting 
the step and the surrounding pixels (for the 

characteristics differences in (1) above) become large and, 
as a result, it is rather difficult to distinguish the 
25 image structure of the pixel at which j aggies have 
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occurred from the original one even when characteristics 
differences among the pixel manifesting the step and the 
surrounding pixels are detected. 

Accordingly, attention is paid to adjacent lines 
5 adjacent to a range (location) at which the j aggies have 
occurred and extending in various directions. Among 
these adjacent lines, an adjacent line extending almost 
parallel to the boundary of an inclined line or an 
inclined edge (hereafter referred to as an "adjacent line 

10 along an inclining direction") passes through the inside 
(or the outside) of the boundary of the inclined line or 
the inclined edge within a local area. On the inside of 
such a boundary, which is not greatly affected by j aggies 
directly on the boundary and does not manifest a great 

15 difference in the density of the image structure, good 
pixel continuity is sustained. As a result, the 
characteristics differences among pixels are small on an 
adjacent line extending along the inclining direction, and 
the similarity along the inclining direction is judged to 

20 be high. 

However, since an adjacent line extending along the 
direction other than the inclining direction intersects 
the boundary line with the inclined edge or the inclined 
line or extends directly astride the position at which the 
25 j aggies have occurred, the pixel continuity is interrupted 
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noticeably. For this reason, great characteristics 
differences occur between pixels on an adjacent line 
extending along a direction other than the inclining 
direction. As a result, it is judged that the pixel 
5 similarity is low in an adjacent line extending along a 
direction other than the inclining direction. 

In conformance to the results of a judgment made with 
regard to pixel similarity in this manner, the weighting 
rate along the inclining direction is set especially high. 

10 As a result, the pixel value is smoothed along the 
inclining direction for a pixel at which steps have 
occurred to reduce j aggies. Through such a smoothing 
operation implemented along the inclining direction, the 
image structure of the inclined edge or the inclined line 

15 can be sustained (any line disrupted by j aggies is 
repaired) . 

Thus, through the similarity judgement implemented by 
using the characteristics differences in (2) above, any 
jaggies in an image can be effectively removed and an 
20 image structure that has been disrupted by the jaggies can 
be repaired. 

© Similarity judgment implemented by using 
characteristics differences in both (1) and (2) 
In this case, the degrees to which the 
25 characteristics differences in (1) and (2) are to be 
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reflected in the similarity judgement can be adjusted as 
appropriate. As a result, by appropriately adjusting the 
effects of (1) and (2) described above, the noise in the 
image and the j aggies in the image can be removed in a 
5 well-balanced manner. 

When calculating the characteristics differences 
among the pixels as the absolute values of the differences 
among the pixel values of the plurality of pixels in the 
local area, similarity is judged as explained below in the 

10 similarity judging step. 

First similarity values ttl, yyl , nul and nsl along 
at least four directions are calculated by using the 
absolute values of the differences in either (1) or (2) or 
the absolute values of the differences in both (1) and (2) . 

15 The smaller the first similarity value, the higher the 
degree of similarity between the pixels, and the larger 
the first similarity value, the lower the degree of 
similarity between the pixels. Next, by averaging or 
taking the median of the first similarity values ttl, yyl, 

20 nul and nsl along at least four directions, a 

representative value ml of the first similarity values is 
calculated. Then, based upon the first similarity values 
ttl, yyl, nul and nsl along at least four directions and 
the representative value ml, the following nonlinear 

25 conversion is executed to calculate second similarity 
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values tt2, yy2 , nu2 and ns2 . 
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(5 and y in the expressions above are each a predetermined 
of value which may be 0) 

10 The larger the second similarity value, the higher 

the degree of similarity between the pixels, and the 
smaller the second similarity value, the lower the degree 
of similarity between the pixels. By using the second 
similarity values thus calculated, direction-dependent 

15 low-pass filtering may be implemented as described below. 

In correspondence to the calculated second similarity 
values tt2, yy2 , nu2 and ns2, the weighting rates for 
nearby pixels positioned along at least four directions 
are determined and the weighted pixel values of the nearby 

20 pixels around the target pixel, which correspond to the 
weighting rates thus determined are added to the pixel 
value of the target pixel and the result of addition is 
divided by the sum of the weights for weighted averaging. 
The following is an explanation of features (A) and 

25 (B) of the nonlinear conversion expressed in (expressions 
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1 ~ 4) . 

(A) It is a relative conversion based upon the 
representative value ml 

The representative value ml is determined as an 
5 average of the first similarity values for each target 

pixel (a normal average value obtained by arithmetic means, 
an average of squares, a median or a value obtained 
through another averaging method) . In (expressions 1 ~ 
4) , the differences between the representative value ml 

10 and the respective first similarity values ttl, yyl , nul 
and nsl are calculated in the first comparison terms in 
the max operations. These differences are relative to 
the representative value ml. In order to simplify the 
explanation, the case in which 5=0 and y = 0 in 

15 (expressions 1-4) is considered. When the degree of 
similarity between the pixels along a given direction is 
much higher than the degrees of similarity along the other 
directions, i.e., when the first similarity value along 
the given direction is small and the first similarity 

20 values along the other directions are large, the 

representative value ml assumes an intermediate value 
among the first similarity values, and while the second 
similarity value determined in correspondence to the 
difference along the direction in which the degree of 

25 similarity is much higher is positive and the differences 
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along the other directions are negative, 0 is assumed for 
the second similarity values along the other directions 
through comparison with 0. 

Thus, unlike in a method which uses the reciprocal of 
5 the first similarity value as the second similarity value, 
if 0 is taken as the second similarity value along a 
direction of pixel arrangement in which the degree of 
similarity is lower than a specific threshold value, it is 
possible to set the contribution factor along the 

10 direction at 0 . 

It is to be noted that a case in which 5=0 and y = 
0 is described above in order to simplify the explanation. 
However, by using appropriate numerical values for 5 and y, 
the freedom in varying the threshold value for setting the 

15 second similarity value to 0 can be afforded to increase 
the degree of freedom with regard to the algorithm for 
optimizing the similarity judgement. 

(B) Namely, nonlinear conversion (limiter) is implemented 
by using the threshold value (ml + 5 - y) as the upper 
20 limit. 

When a first similarity value exceeds the threshold 
value (ml + 6 - y) , the first comparison term in the max 
operation becomes smaller than y. In this case, the 
results of calculation through (expressions 1-4) are 
25 limited by the value of y. Consequently, the degree to 
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which a "first similarity value equal to or exceeding the 
threshold value" that appears to be attributable to the 
image structure or the like is reflected in the weighting 
rate can be limited at y- 
5 Preferably, the value of y should be set at 0 to set 

the weighting rate along the direction in which the degree 
of similarity is particularly low apparently due to the 
image structure to 0 . In this case, the contribution 
factor along the direction in which the degree of 
10 similarity is low is excluded and low-pass filtering is 
performed only along the directions manifesting a high 
degree of similarity, to sustain the image structure 
reliably. 

In addition, setting a negative value for y results 
15 in high pass filtering processing (a negative value set 

for the weighting rate) being performed along a direction 
manifesting a low degree of similarity apparently due to 
the image structure, so that the image contrast is 
emphasized along the direction. 
20 As explained in (A) and (B) above, by implementing 

the nonlinear conversion as expressed in (expressions 1 ~ 
4), the weighting rates in the low-pass filtering 
processing can be determined with a higher degree of 
suitability. \ 
25 If the image data are color dSnage data, a similarity 
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judgment is performed based upon at least two types of 
color information in the color image data in the 
similarity judging step. As a result, it is possible to 
make an accurate judgment with regard to a color boundary 
5 at which no difference may manifest based upon one type of 
color information, to\execute correct direction-dependent 
low-pass filtering processing. In other words, it 
becomes possible to make\an accurate judgment with regard 
to a location at which it Ys difficult to judge the image 
10 structure, such as at a color boundary having a uniform 

brightness level. In addition, noise and j aggies can be 
minimized while fully preserving a color boundary with a 
uniform brightness level and theVLike. 

In image processing through which a similarity 
15 judgment is performed based upon at least two types of 

color information in the color image data, the color image 
data which have not yet undergone interpolation processing 
may be used in the similarity judging step, to eliminate 
any risk of making an erroneous judgment on the 
20 directionality of the image structure on account of a 
color artifact component or the like that may manifest 
after the interpolation processing. In addition, since 
the similarity judgement can be performed using the 
information prior to the G full surface interpolation, a 
25 scale of the processing can be reduced. 
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Furthermore, in the image processing through which a 
similarity judgement is performed based upon at least two 
types of color information in the color image data, the 
first similarity value along a specific direction may be 
5 calculated by using the characteristics differences among 
a plurality of same color pixels and/or among a plurality 
of different color pixels along the specific direction to 
make a judgment with regard to the degree of similarity in 
correspondence to the calculated first similarity value. 
10 The following is an explanation of the different 

situations . 

® First similarity values calculated by using 
characteristics differences among same color pixels alone 
Since the characteristics differences among same 

15 color pixels are used, the directionality of the image 

structure can be determined with accuracy in a local area 
manifesting a high degree of saturation. 
© First similarity values calculated by using 
characteristics differences among different color pixels 

2 0 alone 

Since the characteristics differences among the 
pixels that are closest to each other can be obtained, the 
directionality of an even finer image structure can be 
determined. However, it should be kept in mind that this 
25 method may not yield accurate results with regard to the 
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degree of similarity in a local area manifesting a high 
degree of saturation. 

© First similarity values calculated by using 
characteristics differences among same color pixels and 
5 between different color pixels 

In this case, the degrees to which the two types of 
characteristics differences are reflected in the 
similarity judgement can be adjusted as appropriate. As 
a result, the directionality of the image structure can be 

10 determined in an optimal manner by combining the 
advantages of the two methods described above . 

When calculating the characteristics differences 
among different color pixels, color image data having 
undergone white balance processing may be used. By 

15 performing white balance processing on color image data, 

an extreme color bias in the color image data is prevented 
from occurring. Thus, the risk of color bias manifesting 
as a characteristics differences is reduced to enable an 
accurate judgment with regard to the directionality of the 

20 image structure. 

In addition, when calculating characteristics 
differences among different color pixels, the saturation 
of the target pixel undergoing low-pass filtering 
processing may be detected and the contribution factor of 

25 the characteristics differences among a plurality of a 
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different color pixels may be varied in correspondence to 
the detected saturation in the similarity judging step. 
Generally speaking, large characteristics differences 
among different color pixels at a location manifesting an 
5 extremely high degree of saturation tends to be 

erroneously judged to be the directionality of the image 
structure. Thus, by varying the contribution factor of 
the characteristics differences among different color 
pixels in correspondence to the degree of saturation, the 

10 directionality of the image structure can be determined 
with a higher degree of accuracy. 

When performing image processing on image data having 
undergone interpolation processing on pixels that have 
missing color components, low-pass filtering processing is 

15 performed only on pixels having undergone the 

interpolation processing in the direction-dependent low- 
pass filtering step. 

Under normal circumstances, color artifact or j aggies 
attributable to interpolation processing may occur. The 

20 image processing method according to the present invention 
is effective in reducing such color artifact and j aggies. 
Accordingly, the low-pass filtering processing according 
to the present invention is specifically implemented on 
the pixels that have undergone the interpolation 

25 processing and thus are likely to manifest color artifact 
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or j aggies. As a result, color artifact and j aggies 
attributable to the interpolation processing can be 
reduced with a high degree of reliability. 

Pixels that have ndt been interpolated, on the other 

\ 

5 hand, are not subject to any low-pass filtering processing 
(or the low-pass f iltering\processing is performed to a 
lesser degree) . Consequently, data corresponding to 
pixels that have not been interpolated are preserved in 
their original form. Thus, the user is able to implement 

0 desired image processing later on based upon the original 
data corresponding to pixels that\have not been 
interpolated. \ 

When performing image processing of the image data 
that have undergone interpolation processing on pixels 

5 having missing color components, it is desirable to limit 
the pixel value of each pixel that has undergone 
interpolation processing at a cutoff value corresponding 
to the largest pixel value or the smallest pixel value in 
a specific area near the pixel having undergone 

0 interpolation processing prior to the low-pass filtering 
processing in the similarity judging step. 

A peak may form at an interpolated pixel in a special 
type of interpolation processing. Such a peak at the 
interpolated pixel is limited prior to the low-pass 

5 filtering processing. As a result, the peak at the 



17 




interpolated pixel is prevented from spreading to nearby 
pixels through the low-pass filtering processing to 
achieve an improvement in the image quality. 

When performing image processing on color image data 
5 having, at least, a first color with a highest pixel 

density and a second color with a low pixel density and 
vacancies of color component, the image processing 
procedure comprises a color difference calculating step in 
which the color difference between the second. color and 

10 the first color is obtained for each pixel at which the 

second color is present, a color difference interpolating 
step in which a color difference interpolation value is 
obtained for a pixel at which the second color is not 
present based upon the color difference obtained in the 

15 color difference calculating step and a second color 

restoring step in which the second color is restored based 
upon the color difference interpolation value obtained in 
the color difference interpolating step and the first 
color. In this case, the first color used in the 

20 calculation of the color difference in the color 

difference calculating step is the first color in the 
image data that has not undergone the low-pass filtering 
processing . 

In the image processing method described above in 
25 which the first color in the image data that has not 
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undergone the low-pass filtering processing is used to 
calculate the color differences, the color differences are 
obtained by using a large volume of information available 
prior to the low-pass filtering processing and 
5 interpolation processing can be implemented to obtain data 
as close to those for the original image as possible based 
upon the color differences obtained by using the large 
volume of information. 

It is desirable to use the first color in the image 

10 data having undergone the low-pass filtering processing 
for the restoration of the second color in the second 
color restoring step. Through the low-pass filtering 
processing, noise and jaggies in the first color have been 
suppressed to a sufficient degree. By adding the first 

15 color having undergone the low-pass filtering processing 

to the color difference after the interpolation processing 
(or by multiplying the color difference by the first 
color) the second color is generated. In this case, 
noise and jaggies spreading to the second color via the 

2 0 first color are reduced so that the subsequent image 

processing (such as low-pass filtering processing) on the 
second color can be omitted or can be performed only to a 
lesser degree. 

In image processing performed on color image data 

25 having, at least, a first color with the highest pixel 
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density and a second color with a low pixel density and 
vacancies of color component, weighting rates are obtained 
using the first color and weighted averaging on the second 
color or color difference are performed using the 
5 weighting rates in the direction-dependent low-pass 
filtering step. Since the first color achieves the 
highest pixel density, the weighting rates ascertained in 
conformance to the first color reflect structures in the 
image with the highest degree of accuracy. By using such 

10 weighting rates while performing low-pass filtering 

processing of the second color (or the color difference) , 
accurate low-pass filtering processing is implemented on 
the second color. In addition, in such a case, the 
weighting rates obtained for the first color low-pass 

15 filtering processing can be utilized. Thus, no weighting 
rates need to be calculated specifically for the second 
color to achieve a reduction in the length of time 
required for the image processing. 

In the direction-dependent low-pass filtering step, 

20 the weighting rate along, at least, the direction in which 
the degree of similarity is the lowest can be set 
essentially to 0 . 

By setting the weighting rate along the direction 
manifesting the least similarity essentially to 0, the 

25 image structure along the direction can be preserved in a 
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reliable manner and disintegration of the image, which may 
otherwise occur during low-pass filtering processing, can 
be reliably prevented. 

v , 

When performing image processing oil itrtn-gn rintn ■ 

generated by performing color separation on reflected 
light originating from the object of ipct^ge capturing with 
color filters arranged in a Bayer/array, a judgment is 
made on the similarity manifesting in original image data 
constituted of color separated R, G and B pixel data 
(referred to as original image data) , G color image data 
generated by using/the original image data or brightness 
image data generated from the original image data in the 
similarity judging step. In this case, in the low-pass 
filtering syep, low-pass filtering is implemented on the G 
15 color imag'e data generated from the original image data, R 
color image data generated from the original image data, B 
color image data generated from the original image data, 
the brightness image data generated from the original 
imag4 data or color differences between individual RGB 
20 coyors image data generated from the original image data, 
based upon the Jsimilarity ascertained in the similarity 
j^d^igoftg ^ste p^r 

The present invention may also be adopted in an image 
capturing apparatus and an electronic camera that 
25 implements, at least, low-pass filtering on image data. 
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The image processing apparatus and the electronic camera 
comprise a means for similarity judgment that judges 
degrees of similarity between pixels along at least four 
directions in a local area containing a target pixel 
5 undergoing low-pass filtering processing and a means for 
direction-dependent low-pass filtering that performs 
weighted averaging in which weighted pixel values of the 
pixels around the target pixel are added to the pixel 
value of the target pixel and the result of addition is 

10 divided by the sum of the weights, the weighting rate 

along a direction manifesting a high degree of similarity 
becoming increased based upon the results of the judgment 
performed by the means for similarity judgment. 

The present invention may also be adopted in a 

15 recording medium that records an image processing program 
for implementing, at least, low-pass filtering on image 
data. Moreover, the present invention may be adopted in 
a data signal used to transmit the image processing 
program via a transmission line. The recording medium 

20 according to the present invention may be a recording 

medium (e.g., a memory card) that is detachably mounted at 
the portable information apparatus such as an electronic 
camera. In such a case, the portable information 
apparatus reads out and then executes the image processing 

25 program stored in the recording medium mounted at the 
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apparatus . 

BRIEF DESCRIPTION OF THE DRAWINGS 
FIG. 1 shows the image processing program in a first 
5 embodiment ; 

FIG. 2 illustrates the calculation process performed 
to obtain first similarity values; 

FIG. 3 presents photographs of halftone images on the 
display; 

10 FIG. 4 is a block diagram illustrating the structure 

of the electronic camera 11; 

FIG. 5 is a flowchart of the image processing program 
in a second embodiment; 

FIG. 6 illustrates the operating procedure adopted in 
15 the image processing program; 

FIG. 7 is a flowchart of an image processing 
operation performed in the electronic camera; and 
FIG. 8 is a flowchart of an image processing 
operation performed in the electronic camera. 

20 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 
The following is an explanation of the preferred 
•embodiments of the invention given in reference to the 
drawings . 
25 (First Embodiment) 
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The image processing method in the first embodiment 
is implemented by executing an image processing program on 
a computer. If the image processing program is recorded 
in a recording medium such as a CD-ROM, the image 
5 processing program is copied onto a hard disk at the 

computer from the recording medium set at the computer for 
use. Alternatively, the image processing program may be 
downloaded onto the hard disk at the computer via a 
communication network such as the Internet for use. 
10 E JG -= — 1 — ±& a riowchaiL u£ Lhu - n ne rat ing procedur e 
this image processing program. The tpJ-dr5w±ng is an 
explanation of the operation p§^formed in the image 
processing in the first ejptfodiment given in reference to 
the operating procedure shown in FIG. 1. First, the 
15 computer reads oubr an image file to undergo the processing, 
and opens and axores the image data to undergo the 
processing In memory (step SI) , The image processing is 
performed yon single -color image data opened in the memory 
in this fanner. The single-color image data include 
20 image d&ta corresponding to each color in a color image 

and b/ightness image data synthesized from a plurality of 
colons. The computer selects a target pixel FG[i, j] to 
untifergo the processing located at an ith row and a jqh 
cc/lumn ^jua^-tlife image "dat^and sets a local area ranging 
25 Offer approximately five rows aM<ive columns aj?o1ind the 
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t*>ers?get pixel (step S2) . The image data — to- 
processing are image data corressperrtxTing to a plurality of 
pixels over n rows and m^dolumns , and data corresponding 
to each pixel havp^a signal level referred to as a pixel 
value. Thu&f the term "pixel level" refers to the signal 
level representing the pixel value. It is to be noted 
that yche term "target pixel" does not specif icallyj refer 
tc 



"actual pixel at the ima^e^capturingeli 
The computer performs a calculation using the 
expressions below based upon the level differences among 
the pixels within the local area mentioned above, and 
obtains first similarity values ttl, yyl , nul and nsl 
along four directions (vertical, horizontal, upward 
diagonal and downward diagonal) (step S3) . It is to be 
noted that the vertical direction refers to the direction 
along which the columns of image data provided over n rows 
and m columns extend and the horizontal direction refers 
to the direction in which the rows of the image data 
provided over n rows and m columns extend. 

First similarity value ttl along the vertical direction: 
ttl = ax ( | FG [ i , j] - FG[i, j - 1] | + |FG[i, j] - 

FG[i, j +1] | ) 

+ b x (|FG[i, j + 1] - FG [i , j - 1] | + | FG [i - 1, j] 
- FG[i - 1, j + 2] | 
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+ |FG[i - 1, j] - FG[i - 1, j - 2] | ) 

+ c x ( | FG [ i , j + 1] - FG [ i , j - 1] | + | FG [i + 1, j]' 

- FG [i + 1, j + 2] | 

+ | FG [i + 1, j] - FG [i + 1, j - 2] |) 
5 ... (expression 5) 

First similarity value yyl along the horizontal direction: 
yyl = ax ( | FG [ i , j] - FG [i - 1, j]| + | FG [ i , j] - 

FG [i + 1, j] | ) 

10 + b x ( | FG [ i + 1, j] FG [i - 1, j] | + | FG [ i , j - 1] - 

FG [ i + 2 , j - 1] | 

+ | FG [ i , j - 1] - FG [ i - 2, j - 1] |) 

+ c x ( | FG [ i + 1, j] - FG [i - 1, j] | + | FG [ i , j + 1] 

- FG[i + 2, j + 1] | 

15 + | FG [ i , j + 1] - FG [i - 2, j + 1] |) 

. . . (expression 6) 

First similarity value nul along the upward diagonal 
direction : 

20 nul = ax ( | FG [ i , j] - FG [i +1, j - 1] | + | FG [ i , j] - 

FG[i - 1, j + 1] | ) 

+ b x ( | FG [ i - 2, j + 1] - FG [ i - 1, j]| + | FG [ i - 1, 
j] - FG [i , j - 1] | 

+ | FG [ i , j - 1] - FG[i + 1, j - 2]|) 
25 + c x ( | FG [ i - 1, j + 2] - FG [i , j + 1] | + | FG [ i , j + 
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1] - PG[i + l, j ] | 

+ |FG[i + 1, j] - FG[i + 2, j - 1] I ) 
. . . (expression 7) 

5 First similarity value nsl along the downward diagonal 
direction : 

nsl = ax ( | FG [ i , j] - FG[i - 1, j - 1]| + |FG[i, j] - 

FG[i + 1, j + 1] | ) 

+ b x (|FG[i - 2, j - 1] - FG [i - 1, j] | + |FG[i - 1, 
10 j] - FG[i, j + 1] | 

+ |FG[i, j + 1] - FG [i + 1, j + 2]|) 

+ c x (|FG[i - 1, j - 2] - FG[i, j - 1] | + | FG [ i , j - 
1] - FG [i + 1, j ] | 

+ |FG[i + 1, j] - FG [i + 2, j + 1]|) 
15 ... (expression 8) 

FIGS. 2A - 2D respectively illustrate the operation 
expressions in (expressions 5-8) above. In FIGS. 2A - 
2D, each two-ended arrow indicates the position of the 

20 pixel to undergo a level difference calculation and a or b 
on the arrow indicates a level difference proportional 
constant. It is to be noted that the term of a 
proportional constant c which is not shown, is at a 
position that achieves a symmetrical relationship with the 

25 term of the proportional constant b relative to the target 
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pixel . 

The computer performs the following averaging 
operation on the first similarity values along the four 
directions to calculate a representative value ml (step 
5 S4) . 

ml = (ttl + yyl + nul + nsl) / 4 
... (expression 9) 

10 Next, the computer performs the following nonlinear 

conversion based upon the first similarity values ttl, yyl/ 
nul and nsl along the four directions and the 
representative value ml to calculate second similarity 
values tt2, yy2 , nu2 and ns2 along the four directions 

15 (step S5) . 

tt2 = max {ml - ttl + 5, y} ... (expression 10) 

yy2 = max {ml - yyl +5, y} ... (expression 11) 

nu2 = max {ml - nul +6, y} ... (expression 12) 

20 ns2 = max {ml - nsl +5, y} ... (expression 13) 
with 5=0 and y = 0, for instance. 

As (expressions 5-8) indicate, the absolute value 
of the difference between the pixel values being compared 
25 is defined as a first similarity value, and consequently, 



28 




the smaller the first similarity value, the higher the 
degree of similarity between the pixels and the larger the 
first similarity value, the lower the degree of similarity 
between the pixels. Thus, the first similarity value 
5 might be referred to as a non-similarity value instead. 
In contrast, as (expressions 10-13) indicate, the 
difference between the representative value of the first 
similarity values and the corresponding first similarity 
values is defined as a second similarity value and 

10 consequently, the larger the second similarity value, the 
higher the degree of similarity between the pixels and the 
smaller the second similarity value, the lower the degree 
of similarity between the pixels. Thus, the second 
similarity value indicates the degree of similarity. 

15 It is to be noted that the degree of similarity 

between individual sets of pixel data may be judged by 
multiplying the pixel values of the plurality of pixels. 
In such a case, the larger the value resulting from the 
multiplication, the higher the degree of similarity. 

20 The computer normalizes the second similarity values 

tt2, yy2 , nu2 and ns2 along the four directions to obtain 
weighting rates tt, yy, nu and ns along the four 
directions (step S6) . 

25 m2 = tt2 + yy2 + nu2 + ns2 ... (expression 14) 
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tt = tt2 / m2 ... (expression 15) 
YY = yy2 / m2 . . . (expression 16) 
nu = nu2 / m2 ... (expression 17) 
ns = ns2 / m2 ... (expression 18) 

5 

The computer performs weighted averaging in which the 
weighted pixel values of neighboring pixels are added to 
the pixel value of the target pixel FG[i, j] and the 
result of addition is divided by the sum of the weights to 
10 calculate a target pixel G[i, j] having undergone the low- 
pass filtering processing through (expression 19) (step 
S7) . 

G[i, j] = { FG [ i , j] 
15 + k x tt (FG[i, j - 1] + ( FG [ i , j + 1] ) 

+ k x yy (FG [i - 1, j] + ( FG [ i + 1, j]) 
+ k x nu (FG [i - 1, j + 1] + (FG [i + 1, j -1]) 
+ k x ns ( FG [i - 1, j - 1] + (FG [i + 1, j + 1])} / (1 
+ k x 2) 
20 ... (expression 19) 

With k representing a constant value used to adjust the 
extent of the low-pass filtering processing, which may be, 
for instance, 1/2. 
25 The computer records the pixel G[i, j] having 
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undergone the low-pass filtering processing in a memory 
area for recording the low-pass filtering processing 
results provided in the memory (step S8) . At this point, 
the computer makes a decision as to whether or not the 
5 low-pass filtering processing has been completed for all 
the pixels (step S9) . If it is decided that the low-pass 
filtering processing has not been completed (step S9 NO) , 
the computer selects a new target pixel in the image data 
(step S10) and the operation returns to step S2 explained 

10 earlier. If, on the other hand, it is decided that the 
low-pass filtering processing on all the pixels has been 
completed (step S9 YES) , the computer converts the image 
data having undergone the low-pass filtering processing to 
an image file in a specific format, for instance, before 

15 ending the operation. 

Advantages achieved by performing the image 
processing in the first embodiment described above are now 
explained in reference to FIGS. 3A - 3E. FIGS. 3A ~ 3B 
present photographs of halftone images on the display. 

20 FIG. 3A shows original image data which have not yet 

undergone the low-pass filtering processing. It is to be 
noted that in the figure, the fine structure of the image 
and the locations at which j aggies A and B have occurred 
are indicated by encircling them with white dotted lines. 

25 FIG. 3B shows image data having undergone non- 
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direction-dependent low-pass filtering processing. It is 
to be noted that this low-pass filtering processing is 
local averaging operation processing performed by setting 
the weighting coefficient for the target pixel at 1/2 and 
5 the weighting coefficients for the surrounding 8 pixels 
uniformly at 1/16. In this case, the noise in the 
overall image becomes reduced. However, the definition 
of the image becomes lowered and there is a significant 
loss of the fine structure of the image. In addition, 

10 the j aggies A and B are still present and the jaggedness 
on the inclining lines is noticeable. 

FIG. 3C shows image data having undergone the low- 
pass filtering processing in the embodiment performed by 
setting the proportional constants at a = 1 and b = c = 0 

15 in the first similarity value calculating expressions (5 ~ 
8) . In this low-pass filtering processing, the first 
similarity values are calculated mainly by using the pixel 
level differences on the central lines Lc passing through 
the target pixel, as shown in FIGS. 2A - 2D. Through the 

20 low-pass filtering processing, the noise in the overall 
image becomes reduced. In addition, the definition of 
the image does not become lowered and the fine structure 
of the image is well preserved. Furthermore, a degree of 
improvement is observed in the j aggies B. However, the 

25 j aggies A are still present and the steps on inclining 
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lines are noticeable. 

FIG. 3D shows image data having undergone the low- 
pass filtering processing in the embodiment performed by 
setting the proportional constants at b= 1 and a = c = 0 
5 in the first similarity value calculation expressions (5 ~ 
8) . In this low-pass filtering processing, the first 
similarity values are calculated mainly by using the pixel 
level differences on adjacent lines Ln passing adjacent to 
the target pixel, as illustrated in FIGS. 2A - 2D. Such 

10 low-pass filtering processing reduces the noise in the 
entire image. The definition of the image, however, 
becomes somewhat reduced and there is a slight loss in the 
fine structure of the image. A pronounced improvement is 
observed in the j aggies A and B, however, and the steps on 

15 the inclining lines are broken up (smoothed) in small 
pixel units and are much less noticeable. 

FIG. 3E shows image data having undergone the low- 
pass filtering processing in the embodiment performed by 
setting the proportional constants at a = b = 1 and c = 0 

20 in the first similarity value calculation expressions (5 ~ 
8) . In this low-pass filtering processing, the first 
similarity values are calculated based upon both the 
"pixel level differences on the central lines Lc" and the 
"pixel level differences on the adjacent lines Ln, 11 as 

25 shown in FIGS. 2A - 2D. Such low-pass filtering 
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processing reduces the noise in the overall image. In 
addition, the definition of the overall image is not 
lowered and the fine structure of the image is well- 
preserved. Furthermore, a marked improvement is observed 
5 in the j aggies A and B and the steps on the inclining 

lines are smoothed in small pixel units and are much less 
noticeable . 

While the first similarity values are calculated 
based upon the level differences among pixels in the first 

10 embodiment, the present invention is not limited to this 

example. Generally speaking, the first similarity values 
only need to be calculated based upon differences in the 
characteristics of individual pixels. For instance, when 
processing a color image, the first similarity values may 

15 be calculated based upon hue differences, saturation 

differences or the like. The first similarity values may 
also be calculated by extracting the differences in the 
characteristics in an image obtained through filtering 
processing performed on image data. 

20 In addition, while the first similarity values are 

calculated using (expressions 5-8) in the embodiment, 
the present invention is not limited to this example. 
Under normal circumstances, central lines Lc passing 
through the target pixel and adjacent lines Ln extending 

25 adjacent to the target pixel are set within a local area 
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containing the target pixel. Then, the differences in 
the characteristics of adjacent pixels or pixels 
positioned over a distance with a single pixel present 
between them on these lines Lc and Ln may be ascertained 
5 to calculate the first similarity values by adding these 
characteristics differences at a specific ratio. 

For instance, the first similarity values ttl and yyl 
may be calculated using (expressions 20 and 21) below. 

10 First similarity value ttl along the vertical direction: 

ttl = ax (|FG[i, j] - FG[i, j - 1] | + |FG[i, j] - 

FG[i, j + 1] | ) 

+ b x (|FG[i, j - 1] - FG[i - 1, j + 2] | + | FG [ i - 1, 
15 j] - FG[i - 1, j - 2] | ) x s 

+ c x ( | FG t i + 1 / j] - FG[i + 1, j + 2] | + | FG [ i + 1 , 
j] - FG[i + 1, j - 2] | ) x s 
. . . (expression 20) 

20 First similarity value yyl along the horizontal direction: 

yyl = ax ( | FG [ i , j] - FG[i - 1, j]| + | FG [ i , j] - 

FG [ i + 1 , j] | ) 

+ b x ( | FG [ i , j - 1] - FG [i + 2, j - 1]| + |FG[i, j - 
25 1] - FG[i - 2, j- 1] | ) x s 
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+ c x ( I FG [ i , j + 1] - FG[i + 2, j + 1] | + |FG[i, j + 
1] - FG [i - 2, j + 1] | ) x s 
. . . (expression 21) 

5 with s representing a coefficient used to make up for the 
smaller number of the absolute value addition terms 
compared to that along the diagonal directions, and which 
may be set at, for instance, 1 ~ 1.5. 

In addition, while the second similarity values are 
10 calculated by using (expressions 9-13) in the embodiment, 
the present invention is not limited to this example. 
For instance, instead of (expressions 9 ~ 13), 
(expressions 101 ~ 107) below may be used to calculate the 
second similarity values. 

15 



ty = 


ttl - 


■ yyi • • 




(expression 101) 


us = 


nul - 


- nsl . . , 




(expression 102) 


tt2 


= max 


{ " ty, 


0} 


. . . (expression 103) 


yy2 


= max 


{ty, 0} 




(expression 104) 


nu2 


= max 


{ - us, 


0} 


. . . (expression 105) 


ns2 


= max 


{us, 0} 




(expression 106) 



if tt2 + yy2 + nu2 + ns2 = 0 
then tt2 = 1, yy2 = 1, nu2 = 1, ns2 = 1 
25 ... (expression 107) 
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In (expressions 101 - 106) , the following two types of 
processing 1 and 2, which are independent of each other, 
are performed at once : 
5 1. The weighting rate in the direction-dependent 

low-pass filtering processing is set substantially to 0 
for the direction manifesting a lesser degree of 
similarity in each pair of directions such as "vertical 
and horizontal" or "upward diagonal and downward diagonal" 

10 extending roughly perpendicular to each other. 

2. The weighting rate in the direction-dependent 

low-pass filtering processing is determined based upon the 
"absolute value of the difference between the first 
similarity values along the two directions" for the 

15 direction manifesting a higher degree of similarity in 

each pair of directions such as "vertical and horizontal" 
or "upward diagonal and downward diagonal" extending 
roughly perpendicular to each other. 

In the processing 1 above, low-pass filtering 

20 processing is executed only along one of the two 

directions extending roughly perpendicular to each other. 
As a result, low-pass filtering processing is not 
implemented along the two directions extending almost 
perpendicular to each other at once, thereby making it 

25 possible to sustain the directionality of the image 
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structure with a higher degree of clarity. 

In the processing 2, on the other hand, the low-pass 
filtering processing is implemented more rigorously along 
the direction manifesting the higher degree of similarity 
5 as the difference in the first similarity values along the 
two directions extending roughly perpendicular to each 
other becomes larger. As a result, it is possible to 
reduce noise and j aggies more thoroughly along a direction 
manifesting pronounced similarity. 

10 Next, another embodiment is explained. 

( Second Embodiment ) 

The second embodiment relates to an electronic camera 
that processes captured images through an image processing 
method similar to that in the first embodiment. FIG. 4 

15 is a block diagram illustrating the structure of an 
electronic camera 11 in the second embodiment. 

In FIG. 4, a taking lens 12 is mounted at the 
electronic camera 11. A light-receiving surface of an 
image capturing element 13 is set in the image space of 

20 the taking lens 12 . RGB image data output by the image 
capturing element 13 are digitized via an A/D conversion 
unit 14, and are then input to an image processing unit 15 
constituted of a microprocessor. A read only memory 17, 
an image memory 18 and the like are connected to the image 

25 processing unit 15 via a data bus 16. An image 
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processing program is recorded in the read only memory 17. 
The image processing unit 15 reads out and executes this 
image processing program. 

FIG. 5 is a flowchart of the operating procedure 
5 adopted in the image processing program. FIG. 6 

illustrates a data flow of the image data at various 
stages during the image processing. The following is an 
explanation of the image processing operation achieved in 
the embodiment, given in reference to FIGS. 4-6. 

10 Image data ORG output by the image capturing element 

13 are data constituted of three colors R, G and B in a 
Bayer array. One of the following pixel values G (O,^), 
R (0,^M and B (O , is assigned to each of the pixels in 
the image data ORG in conformance to the rules of the 

15 Bayer array. (0,^M indicates the row and the column with 
which a given pixel position is specified. 

The image processing unit 15 takes in the image data 
ORG and executes image processing through the following 
procedure. An explanation of the processing performed in 

20 individual steps follows. 

© Step S21: first similarity value calculation for 
interpolation processing 

First, the image processing unit 15 calculates a 
first similarity value CvO along the vertical direction 

25 and a first similarity value ChO along the horizontal 
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direction for a pixel with no G color component, i.e., for 
a vacancy [i, j], through the following expressions (22 ~ 
27) . 

5 Cvl = |G[i, j - 1] - G[i, j + 1] | ... (expression 22) 

Cv2 = (|G[i, j - 1] - Z[i, j]|+ |G[i # j + 1] - Z[i, j]|) / 
2 ... (expression 23) 

Chi = |G[i - 1, j] - G[i +1, j]| ... (expression 24) 
Ch2 = (|G[i - 1, j] - Z[i, j]|+ |G[i + 1, j] - Z[i, j]|) / 
10 2 . . . (expression 25) 

Cv0[i, j] = Al * Cvl + A2 • Cv2 ... (expression 26) 
Ch0[i, j] = Al • Chi + A2 • Ch2 ... (expression 27) 

The value corresponding to either R or B is used for Z. 

15 Al and A2 each represent a weighting coefficient, with at 
least either one of the weighting coefficients being a 
value other than 0. Typically, Al = A2 = 1 . 
© Step S22: first similarity value smoothing processing 
Next, the image processing unit 15 calculates first 

20 similarity values Cv and Ch by calculating neighborhood 
averages for the first similarity values CvO and ChO 
through the following expressions (28 and 29) 

Cv[i, j] =(4 • Cv0[i, j] + CvO [i - 1, j - 1] + CvO [i + 1, 
25 j - 1] 
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+ CvO [i - 1, j + 1] + CvOti + 1, j + 1]) / 8 
(expression 28) 

Ch[i, j] =(4 • ChO[i, j] + ChO[i - 1, j - 1] + ChO [i + 1, 
j - 1] 

5 + ChO [i - 1, j + 1] + ChO[i + 1, j + 1]) / 8 

(expression 29) 

It is to be noted that these neighborhood averages 
may be simplified to; 

10 

Cv[i, j] =CvO[i, j] 
Ch[i, j] =ChO[i, j] 

© Step S23 : judgement of interpolation direction based 
15 upon first similarity values 

Next, the image processing unit 15 sets a direction 
flag HV in conformance to the results of the following 
conditional judgement : 

Vertical similarity judgment: HV[i, j] = 1 if Cv[i, j] + 
20 thO < Ch[i, j] 

Horizontal similarity judgment: HV[i, j] = -1 if Ch[i, j] 
+ thO < Cv[i, j] 

Non-directionality judgment: HV[i, j] = 0 in all cases 
other than the above 
25 thO is a threshold within a range of approximately 0 
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~ several tens if the range that may be assumed for G([i, 

j] is, for instance, 0 ~ 255. 

© Step S24 : G interpolation operation 

The image processing unit 15 executes the following 
5 interpolation operation in conformance to the setting at 
the direction flag HV. 

A pixel with a G component, i.e., in the case of no 
vacancy: FG[i, j] =G[i, j] 

Vertical similarity: FG[i, j] = Gv if HV[i, j] =1 
10 Horizontal similarity: FG [i , j] = Gh if HV[i, j] = -1 

No directionality: FG [ i , j] = (Gv + Gh) / 2, if HV[i, j] = 
0 

Gv and Gh above represent values obtained as follows: 



15 Gv = (G[i, j - 1] + G[i, j + 1] ) / 2 + CI • ( 2 • Z[i, j] 

- Z [i, j - 2] - Z [i, j + 2] ) / 4 

Gh = (G[i - 1, j] + G[i + 1, j] ) / 2 + CI • ( 2 • Z[i, j] 

- Z [i - 2, j] - Z [i + 2, j] ) / 4 



20 with CI representing a specific constant which may be 0 , 1 

or an intermediate value between them. 

© Step S25: R and B color difference conversion 

The image processing unit 15 executes the following 

color difference conversion at the position of each pixel 
25 which originally contains either the R color component or 
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the B color component, in reference to the interpolated G. 

DR[i, j] = R[i, j] - FG[i, j] ... (expression 30) 
DB[i, j] = B[i, j] - FG[i, j] ... (expression 31) 

5 

© Step S26: color difference interpolation 

The image processing unit 15 executes the following 
interpolation processing of the color difference data 
calculated as described above. 

10 

If there are R pixels above and below the target pixel : 
FDR [ i , j] = (DR[i, j - 1] + DR[i, j + 1] ) /2 

If there are R pixels to the left and right of the target 
15 pixel : 

FDR [ i , j] = (DR[i - 1, j] + DR[i + 1, j])/2 

If there are R pixels diagonally adjacent to the pixel 
undergoing processing: 
20 FDR [ i , j] = (DR[i - 1, j - 1] + DR[i - 1, j + 1] + DR[i + 
1, j - 1] + DR[i + 1, j + 1] ) /4 

If there are B pixels above and below the target pixel: 
FDB [ i , j] = (DB [i , j - 1] + DB[i, j + 1] ) /2 

25 
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If there are B pixels to the left and right of the target 
pixel : 

FDB [ i , j] = (DB[i - 1, j] + DB[i + 1, j ] ) /2 

5 If there are B pixels diagonally adjacent to the pixel 
undergoing processing : 

FDB [ i , j] = (DB[i - 1, j - 1] + DB [i - 1, j + 1] + DB[i + 
1 # j - 1] + DB [i + l f j + 1] ) /4 

0 © Step S27: range limitation of G interpolation value 
The image processing unit 15 limits the range of the 

G interpolation value by using the largest value Gmax and 

the smallest value Gmin among eight surrounding pixels. 

If exceeding the largest value: FG[i, j] = Gmax, if FG [ i , 
5 j] > (Gmax + Th4) 

If smaller than the smallest value: FG[i, j] = Gmin, if 

FG[i, j] < (Gmin - Th4) 

It is to be noted that Th4 is a threshold value that 

determines the extent to which the peak of the 
0 interpolation value is to be tolerated. In addition, the 

number of surrounding pixels does not have to be 8 and may 

be any number that is larger than, for instance, 6. 

© Step S28: low-pass filtering processing on G - 

interpolated pixel ^ 
5 The image processing unit 15 ^executes direction- 
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dependent low-pass filtering processing similar to the 
image processing achieved in the first embodiment on the 

y \ 

G-interpolated pixel. In this situation, either of the 
following two types of lowWss filtering processing may 
5 be implemented. * 

(a) Direction-dependent low-pass filtering processing is 
performed on all the G pixels as in the image processing 
in the first embodiment. 

(b) Direction-dependent low-pass filtering processing is 
10 implemented only on G pixels having undergone the 

interpolation processing without executing direction- 
dependent low-pass filtering processing on G pixels that 
have not been interpolated. 

© Step S29: low-pass filtering processing on image data 
15 corresponding to a plane formed through color difference 

interpolation, i.e., low-pass filtering processing on 

color difference interpolated data. 

The image processing unit 15 calculates weighting 

rates tt, yy, nu and ns for a G plane resulting from the 
20 interpolation processing, in a manner similar to that 

adopted in the image processing in the first embodiment. 

At a position corresponding to the pixel for which the 

weighting rates are already ascertained in step S28, the 

weighting rates obtained in step S28 are reused. 
25 The image processing unit 15 executes direction- 
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dependent low-pass filtering processing based upon the 
weighting rates on the color difference interpolated image 
data as expressed below. 



5 Tmp_DR[i, j] = [FDR [i , j] 

+ k x tt (FDR[i, j - 1] + FDR [i , j + 1] ) 

+ k x yy ( FDR [ i - 1, j] + FDR [i + 1, j]) 

+ k x nu ( FDR [ i - 1, j + 1] + FDR [i + 1, j - 1]) 

+ k x ns ( FDR [ i - 1, j - 1] + FDR [i + 1, j + 1] ) ] / 

10 (1 + k x 2) ... (expression 32) 



Tmp_DB[i, j] = [FDB [i , j] 

+ k x tt (FDB[i, j - 1] + FDB[i, j + 1] ) 

+ k x yy ( FDB [ i - 1 , j ] + FDB [i + 1, j]) 

15 + k x nu ( FDB [ i - 1, j + 1] + FDB [ i + 1, j - 1]) 

+ k x ns (FDB [i - 1, j - 1] + FDB [i + 1, j +1])] / 

(1 + k x 2) ... (expression 33) 



with k representing a constant value used to adjust the 
20 extent of the low-pass filtering processing, which may be 
set to, for instance, k = 1/2 . 

After completing the weighted averaging operation 
explained above for the entire image plane, the image 
processing unit 15 executes the following substitution: 
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FDR t i , j ] = Tmp_DR [ i , j ] 
FDB [ i , j ] = Tmp_DB [ i , j ] 

® Step S30: R and B restoration 
5 The image processing unit 15 restores R and B by- 

adding the G plane having undergone the low-pass filtering 
processing to the color difference plane data. 

FR[i, j] = FG[i, j] + FDR[i, j] ... (expression 34) 
10 FB[i, j] = FG [i , j] + FDB [ i , j] ... (expression 35) 

The image processing unit 15 outputs FR[i, j], FG[i, 
j] and FB[i, j] thus obtained to a recording device or the 
like. 

15 The advantages achieved in the electronic camera in 

the second embodiment described above are explained. 

The electronic camera in the second embodiment 
achieves advantages similar to those achieved in the first 
embodiment. In addition, the electronic camera in the 

20 second embodiment realizes the following advantages. 

(1) The image processing unit 15 implements low-pass 
filtering processing on G- interpolated pixels and color 
difference interpolated pixels. As a result, color 
artifact and j aggies attributable to the interpolation 

25 processing can be reduced to a satisfactory degree. In 
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addition, if the image processing unit 15 performs low- 
pass filtering processing on G- interpolated pixels and 
color difference interpolated pixels alone and does not 
perform low-pass filtering processing on pixels that have 
5 not been interpolated, the original image data can be 

preserved faithfully and a reduction in the length of time 
required for the processing is achieved. It is to be 
noted that by performing the low-pass filtering processing 
on all the pixels, a smooth image with j aggies and the 
10 like removed satisfactorily is obtained. 

(2) The image processing unit 15 limits the range of the 
peak of interpolated pixels prior to the low-pass 
filtering processing. Thus, the peak of the interpolated 
pixel becomes less noticeable and, at the same time, the 

15 peaks of the interpolated pixels are prevented from 
spreading to nearby pixels. 

(3) The image processing unit 15 ascertains color 
differences relative to the G color component in the image 
data prior to the low-pass filtering processing when a 

20 large volume of information is available. Thus, a large 
volume of color difference information is obtained to 
enable even more accurate color difference interpolation. 

(4) The image processing unit 15 uses the G color 
component obtained through the low-pass filtering 

25 processing to restore the R (B) color component based upon 



48 




the color difference. 



Thus, a problem of noise and 



j aggies being introduced into the R (B) color component 



via the G color component is prevented. 



(5) The image processing unit 15 ascertains the weighting 
5 rates for the G color component with a large volume of 
information available and performs low-pass filtering 
processing of the R (B) color component by using the 



filtering performed at the weighting rates calculated by 

10 using the R (B) color component data, direction-dependent 
low-pass filtering processing with a higher degree of 
accuracy is achieved. 

It is to be noted that the inventor of the present 
invention has confirmed that the direction-dependent low- 

15 pass filtering processing according to the present 

invention achieves good results regardless of whether the 
gradation conversion .( including the gradation correction) 
is implemented either before or after the interpolation 
processing in the embodiment described above by conducting 

2 0 a subjective evaluation on the image quality. 

It is to be noted that while the image processing 
unit 15 executes the image processing program in the 
embodiment explained above, the present invention is not 
limited to this example. For instance, part of or the 

25 entirety of the image processing described above may be 



weighting rates. 



As a result, compared to low-pass 
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executed by an arithmetic circuit or the like in order to 
improve the processing speed of the image processing unit 
15 . 

In addition, while the low-pass filtering processing 
is executed on the color difference data in the embodiment 
described above,. the present invention is not limited to 
this example. For instance, the low-pass filtering 
processing may be implemented on R (B) color component 
image data restored from the color difference data. In 
this case, too, by utilizing the weighting rates 
determined for the G color component, the low-pass 
filtering processing of the R (B) color component image 
data can be implemented with accuracy. 

-Furthermore , ^ ^ ■ 1 ii" 1 inyr-p aafl filtering pr^ 




I-fe — ars — to be noted that the directional low-pass 
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on any plane. For instance, 
implemented on a brightness plane Y 



Another embodiment is explained below. 
5 (Third Embodiment) 

The image processing method in the third embodiment 
is characterized in that at least two types of color 
information are used in similarity judgment. The 
following is an explanation of the operation performed to 
10 calculate first similarity values in the image processing 
in the third embodiment. 

In the third embodiment, color information 
corresponding to three planes constituted of R, G and B is 
prepared for the calculation of the first similarity 
15 values. It is to be noted that if color information 

corresponding to three planes R, G and B is not available, 
the interpolation processing is performed first to obtain 
color information corresponding to three planes R, G and B, 
In the third embodiment, the first similarity values 
20 for a pixel at an ith row and a j th column are calculated 
by using the color information corresponding to the three 
planes R, G and B thus obtained in the following 
expressions (110 ~ 113). 



25 First similarity value along the vertical direction ttl 
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ttl = ax ( | FG [ i , j] - FG [ i , j - 1] | + | FG [ i , j] - 

FG[i, j + 1] | ) 

+ b x ( | FG [ i , j + 1] - FG [i , j - 1] | + | FG [i - 1, j] 
5 - FG [i - 1, j + 2] | 

+ | FG [i - 1, j] - FG [i - 1, j - 2]|) 

+ al x ( | FR [ i , j] - FR [i , j - 1] | + | FR [ i , j] - FR [ i , 
j + 1] | ) 

+ bl x ( | FR [ i , j + 1] - FR [i , j - 1] | + | FR [ i - 1, j] 
10 - FR [ i - 1, j + 2] | 

+ | FR [ i - 1 , j ] - FR [ i - 1, j - 2] | ) 

+ a2 x ( | FB [ i , j] - FB [i , j - 1] | + | FB [ i , j] - FB [i , 
j + 1] D 

+ b2 x { | FB [ i , j + 1] - FB [i , j - 1] | + | FB [i - 1, j] 
15 - FB [i - 1, j + 2] | 

+ |FB [i - 1, j] - FB [i - 1, j - 2] | ) 
. . . (expression 110) 

First similarity value along the horizontal direction yyl : 

20 

yyl = ax ( | FG [ i , j] - FG[i - 1, j]| + | FG [ i , j] - 

FG [i + 1, j] | ) 

+ b x ( | FG [ i + 1, j] - FG [i - 1, j] | + | FG [ i , j - 1] 
- FG[i + 2, j - 1] | 
25 + I FG [ i , j - 1] - FG [i - 2, j - 1] | ) 
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+ al x ( | FR [ i , j] - FRti - 1, j]| + | FR [ i , j] - FR[i 
+ 1, j] I) 

+ bl x ( | FR [i + 1, j] - FR [i - 1, j] | + | FR [ i , j - 1] 

- FR [i + 2, j - 1] | 

5 + | FR [ i , j - 1] - FR [ i - 2, j - 1] | ) 

+ a2 x ( | FB [ i , j] - FB[i - 1, j]| + | FB [i , j] - FB [ i 
+ 1, j] |) 

+ b2 x ( | FB [ i + 1, j] - FB [i - 1, j] | + | FB [ i , j - 1] 

- FB [i + 2, j - 1] | 

10 + | FB [ i , j - 1] - FB [ i - 2, j - 1] | ) 

. . . (expression 111) 

First similarity value along the upward diagonal direction 
nul : 

15 

nul = ax ( | FG [ i , j] - FG[i + 1, j - 1] | + | FG [ i , j] - 

FG[i - 1, j + 1] | ) 

+ b x ( | FG [ i - 2, j + 1] - FG [ i - 1, j] | + | FG [i - 1, 
j] - FG [i , j - 1] | 
20 + | FG [ i , j - 1] - FG [i + 1, j - 2]|) 

+ al x ( | FR [ i , j] - FR [ i + 1, j - 1] | + | FR [ i , j] - 
FR [i - 1, j + 1] | ) 

+ bl x ( | FR [ i - 2, j + 1] - FR [i - 1, j] | + | FR [i - 1, 
j] - FR [i , j - 1] | 
25 + | FR [ i , j - 1] - FR [ i +1, j - 2]|) 
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+ a2 x (|FB[i, j] - FB [ i + 1 , j - 1] | + | FB [ i , j] - 
FB [i - 1, j + 1] | ) 

+ b2 x ( | FB [ i - 2, j +1] - FB [i - 1, j]| + | FB [ i - 1, 
j] - FB [i, j - 1] | 
5 + | FB [ i , j - 1] - FB [i + 1, j - 2]|) 

. . . (expression 112) 

First similarity value along the downward diagonal 
direction nsl: 

10 

nsl = ax ( | FG [ i , j] - FG[i - 1, j - 1]| + | FG [ i , j] - 

FG[i + 1, j + 1] | ) 

+ b x ( | FG [ i - 2, j - 1] - FG [ i - 1, j] | + | FG [i - 1, 
j] - FG [i , j + 1] | 
15 + | FG [ i , j + 1] - FG [i + 1, j + 2] |) 

+ al x ( | FR [ i , j] - FR[i - 1, j - 1] | + | FR [ i , j] - 
FR [i + 1, j + 1] | ) 

+ bl x ( | FR [ i - 2, j - 1] - FR [i - 1, j] | + | FR [ i - 1, 
j] - FR [i , j + 1] | 
20 + | FR [ i , j + 1] - FR [i + 1, j + 2] | ) 

+ a2 x ( | FB [ i , j] - FB [i - 1, j - 1] | + | FB [ i , j] - 
FB [i + 1, j + 1] | ) 

+ b2 x ( | FB [ i - 2, j - 1] - FB [i - 1, j] | + | FB [i - 1, 
j] - FB[i, j + 1] | 
25 + | FB [ i , j + 1] - FB [i + 1, j + 2] | ) 
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. . . (expression 113) 

It is to be noted that in the expressions above, FR 
(Q,Z±), FG (O,^) and FB (O,^) are the individual values 
5 at the O(th) row and the ZMth) column in the color 

information corresponding to the three planes R, G and B. 
In addition, al, a, a2 , bl, b and b2 represent weighting 
coefficients that determine the contribution factors of 
the R, G and B colors in the first similarity values. 

10 Since the process in which direction-dependent low- 

pass filtering processing is performed based upon the 
first similarity values thus obtained is identical to that 
in the first embodiment or the second embodiment, and its 
explanation is omitted. 

15 By implementing the image processing in the third 

embodiment, too, advantages similar to those achieved in 
the first embodiment and the second embodiment are 
realized. It should be noted as a feature particular to 
the third embodiment that the first similarity values are 

20 calculated by incorporating at least two types of color 

information in color image data. Thus, color boundaries 
can be ascertained with a high degree of accuracy even in 
an image in which the G plane sustains almost perfect 
uniformity and the R plane or the B plain alone changes. 

25 As a result, it becomes possible to remove noise and 
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j aggies correctly while fully preserving such color 
boundaries . 

It is to be noted that while the first similarity 
values are calculated by using R, G and B in the 
5 embodiment described above, the present invention is not 
limited to this example. In a general way, similarity 
judgment should be performed by using at least two types 
of information related to the colors in the image. For 
instance, color difference values, complementary color 
10 values or another type of color values may be used as the 
color information . 

In addition, appropriate image data Y may be first 
synthesized by using at least two types of color 
information in the color image data as expressed in 
15 (expression 114) below and a similarity judgment may be 
performed with regard to the image data Y. 

Y=axG+bxR+cxB ... (expression 114) 

20 It is preferable that the weighting coefficients a, b 

and c in the expression above is set at values 
corresponding to the visual sensitivity of the individual 
types of color information to enable a similarity judgment 
to be made in conformance to the visual sensitivity. For 

25 instance, they may be set at: a = 0.6, b=0.3 and c = 0.1 
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when processing a regular image. As a further 
alternative, they may be set at: a =0.6, b=0.2 and c = 
0.2. Alternatively, the weighting coefficients a, b and 
c may be set at arbitrary values. In addition, when 
5 specifically judging the similarity regarding the R 

component, the weighting coefficient b should be set at 
the largest rate. The direction-dependent low-pass 
filtering performed by using the second similarity values 
thus prepared may be implemented on any plane, and it may 

10 be implemented on the Y plane. 

It is to be noted that while the first similarity 
values are calculated at once using the calculation 
expressions above in the embodiment, the present invention 
is not limited to this example. For instance, a 

15 similarity judgment is performed for color information 
with a high degree of visual sensitivity only, and a 
similarity judgment may be implemented with regard to 
other color information only if the directionality of the 
image structure is unclear from the * degree of similarity 

20 judged in the first judgement. In such an arithmetic 

operation performed over stages, the similarity judgment 
is performed first with regard to the color information 
manifesting a high degree of visual sensitivity and, as a 
result, low-pass filtering processing corresponding to the 

25 visual sensitivity can be executed. In addition, since 
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the similarity can be ascertained through first arithmetic 
operation for the majority of pixels, the overall volume 
of arithmetic processing can be reduced. 

Next, another embodiment is explained. 
5 (Fourth Embodiment) 

The electronic camera in the fourth embodiment is 
explained below. It is to be noted that since the 
structure of this camera is identical to that in the 
second embodiment (see FIG. 4) , an explanation on its 

10 structure is omitted. FIG. 7 is a flowchart of the image 
processing operation performed in the fourth embodiment. 
It is to be noted that this image processing operation is 
achieved by replacing steps S28 ~ S29 in the second 
embodiment (see FIG. 5) with steps S41 ~ S45. The 

15 following is an explanation on the difference from the 
second embodiment . 

© Step S41: calculation of first similarity values 
between same color pixels and different color pixels 
The Bayer array image data ORG read out from the 
20 image capturing element 13 are still present in the image 
memory 18. The image data ORG are color image data 
having undergone white balance processing but yet to 
undergo color interpolation processing. 

The image processing unit 15 calculates first 
25 similarity values in the neighboring area around a target 
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pixel at an ith row and a jth column by executing the 
following arithmetic operation on the image data ORG. It 
is to be noted that the pixel value corresponding to a 
given pixel in the image data ORG is uniformly expressed 
5 as Z (O,^) regardless of which color among R, G and B the 
data correspond to. 

First similarity value along the vertical direction ttl: 

10 ttl = pi x (|Z[i, j] - Z[i, j - 1] | + |Z[i, j] - Z[i, 

j + 1] | ) 

+ P 2 x (|z[i - l, j] - z[i - 1, j - 1] | + | Z [i - 1, 
j] - Z [i - 1, j + 1] | ) 

+ p3 x (|Z[i, j + 1] - Z[i, j - 1] | + |Z[i, j] - Z[i, 
15 j - 2] | + |Z[i, j] - Z[i, j + 2] | ) 

+ p4 x (|z[i - 1, j + 1] - Z[i - 1, j - 1] | + |Z[i - 
1, j] - Z[i - 2, j - 2] 

+ | Z [i - 1, j] - Z [i - 1, j + 2] | ) 
. . . (expression 124) 

20 

First similarity value along the horizontal direction yyl : 

yyl = ql x (|Z[i, j] - Z [i - 1, j]| + |Z[i, j] - Z [i + 

1, j] I) 

25 + q2 x (|Z[i, j - 1] - Z[i - 1, j - 1] | + |Z[i, j - 
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1] - Z [i + 1, j - 1] | ) 

+ q3 x (|Z[i + 1, j] - Z[i - 1, j]| + |Z[i, j] - Z [i 
+ 2, j] | + |Z[i, j] - Z[i - 2, j] | ) 

+ q4 x (|Z[i + 1, j - 1] - Z[i - 1, j - 1] | + |Z[i, j 
5 - 1] - Z[i + 2, j - 1] 

+ |Z[i, j - 1] - Z[i - 2, j - 1] |) 
. . . (expression 125) 

First similarity value along the upward diagonal direction 
10 nul : 

nul = rl x ( | Z [i , j] - Z [i + 1, j - 1]| + |Z[i, j] - 

Z[i - 1, j + 1] | ) 

+ r2 x (|Z[i - 1, j] - Z[i, j - 1] | + |Z[i - 1, j] - 
15 Z[i - 2, j + 1] | ) 

+ r3 x (|Z[i - 1, j + 1] - Z[i + 1, j - 1] | + |Z[i, 
j] - Zli + 2, j - 2] | 

+ | Z [i, j] - Z [i - 2, j + 2] | ) 

+ r4 x (|Z[i - 2, j + 1] - Z[i, j - 1] | + |Z[i - 1, 
20 j] - Z[i + 1, j - 2] |) 
. . . (expression 126) 

First similarity value along the downward diagonal 
direction nsl: 

25 
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nsl = si x (|z[i, j] - Z[i - 1, j - 1] | + | Z [i, j] - 

Z [i + 1, j + 1] | ) 

+ s2 x ( | Z [i + 1, j] - Z[i, j - 1] | + |z[i + 1, j] - 
Z [i + 2, j + 1] | ) 
5 + s3 x (|z[i + 1, j + 1] - Z[i - 1, j - 1] | + |Z[i # 

j] - Z[i - 2, j - 2] | 

+ |Z[i, j] - Z[i + 2, j + 2] |) 

+ s4 x (|Z[i + 2, j + 1] - Z[i, j - 1] | + |Z[i + 1, 
j] - Z [i - 1, j - 2] | ) 

10 ... (expression 127) 

In the expressions above, Z (0,^0 represents one of 
value corresponding to R, G or B at the O(th) row and the 
Zi(th) column in the image data ORG. In other words, Z 
(0,^M represents the G component pixel value at a 

15 position at which a G pixel is located, the R component 
pixel value at a position at which an R pixel is located 
and the B component pixel value at a position at which a B 
pixel is located. In addition, pi - 4 , ql - 4 , rl - 4 
and si - 4 are weighting coefficients that determine the 

20 contribution factors of the individual terms in the first 
similarity values, and may all be set at 1, for instance. 

In addition, the addition of the absolute values may 
be weighted. For instance, the third term in 
(expressions 124) may be weighted as follows. 
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p3 x (|Z[i, j + 1] - Z[i, j - 1] | + u • |z[i, j] - Z[i, j 
- 2] | + v • |Z[i, j] - Z[i, j + 2] |) 

It is to be noted that it is desirable to set u and v 
5 at approximately 1/2 . 

© Step S42: weighting adjustments for first similarity 
values along individual directions 

The image processing unit 15 adjusts the weightings 
for the calculated first similarity values along the 
10 individual directions as indicated in (a) below. 

(a) ttl = ttl x 2, yyl = yyl x 2, nul = nul, nsl = nsl 

Through these weighting adjustments, the first similarity 
15 values along the diagonal directions are set relatively 

low to indicate a high degree of similarity and thus, the 
weighting rates along the diagonal directions are set high 
in the direction-dependent low-pass filtering processing. 
As a result, a marked advantage is achieved in that, for 
20 instance, in an image capturing element manifesting a 

level difference such as a gain difference between an odd- 
numbered column (row) and an even numbered column (row) , 
the noise attributable to the level difference is canceled. 
It is to be noted that if the weighting is not adjusted 
25 along the individual directions, the first similarity 
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values are as indicated in (b) below. 

(b) ttl = ttl, yyl = yyl/ nul = nul, nsl = nsl 

5 In an actual application, it is even more desirable 

to execute an appropriate weighting adjustment in 
correspondence to characteristics (defects) of the image 
data. In more specific terms, a final decision should be 
made as to whether the weighting values in (a) or (b) 
10 above or another set of weighting values are to be used 
through a subjective evaluation test of a test image or 
the like. 

© Step S43 : determining weighting rates 

The image processing unit 15 calculates the 
15 representative value ml by performing the following 
averaging operation on the first similarity values. 

ml = (ttl + yyl + nul + nsl) / 4 ... (expression 128) 

20 The image processing unit 15 implements the following 

nonlinear conversion based upon the first similarity 
values ttl, yyl, nul and nsl along the four directions and 
the representative value ml to calculate second similarity 
values tt2, yy2 , nu2 and ns2 along the four directions. 

25 
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with 


5 and 
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set at, 




for 


instance 5=1 and y = 



Next, the image processing unit 15 normalizes the 
second similarity values tt2, yy2 , nu2 and ns2 along the 
four directions to ascertain weighting rates tt, yy, nu 
10 and ns along the four directions. 

m2 = tt2 + yy2 + nu2 + ns2 . . . (expression 133) 
tt= tt2 / m2 ... (expression 134) 
yy= yy2 / m2 . . . (expression 135) 
15 nu= nu2 / m2 . . . (expression 136) 
ns= ns2 / m2 ... (expression 137) 

® Step S44 : direction-dependent low-pass filtering 
processing for G plane 
20 The image processing unit 15 performs weighted 

averaging by adding weighted pixel values of neighboring 
pixels to the pixel value of the target pixel FG[i, j] and 
then calculates a pixel FG[i, j] after the low-pass 
filtering processing using (expression 138) . 
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Tmp_FG[i, j] = {FG[i, j] 

+ k x tt (FG[i, j - 1] + FG [i , j + 1] ) 
+ k x yy (FG [i - 1, j] + FG [i + 1, j]) 
+ k x nu (FG [i - 1, j + 1] + FG [i + 1, j -1]) 
5 + k x ns (FG [i - 1, j - 1] + FG [i + 1, j + 1])} / (1 

+ k x 2) 

. . . (expression 138) 

with k representing a numerical value used to adjust the 
10 extent of the low-pass filtering processing, which may be 
set at, for instance, k = 1/2. 

© Step S45: low-pass filtering processing on color 
difference interpolated pixel 

The image processing unit 15 executes direction- 
15 dependent low-pass filtering processing of color 

difference interpolated pixels by using the following 
expressions . 

Tmp_DR[i, j] = { FDR [i , j] 
20 + k x tt ( FDR [ i , j - 1] + FDR [ i , j + 1]) 

+ k x yy ( FDR [ i - 1, j] + FDR [i + 1, j]) 
+ k x nu ( FDR [ i - 1, j + 1] + FDR [i + 1, j - 1]) 
+ k x ns ( FDR [ i - 1, j - 1] + FDR [ i + 1, j + 1])} / 
(1 + k x 2) 
25 ... (expression 139) 
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TmpJDBfi, j] = { FDB [ i , j] 

+ k x tt ( FDB [ i , j - 1] + FDB [ i , j + 1] ) 

+ k x yy ( FDB [ i - 1, j] + FDB [i + 1, j]) 

+ k x nu ( FDB [ i - 1, j + 1] + FDB [i + 1, j -1]) 

5 + k x ns ( FDB [ i - 1, j - 1] + FDB [i + 1, j + 1])} / 

(1 + k x 2) 

(expression 140) 

with k representing a constant value used to adjust the 
10 extent of the low-pass filtering processing, which may be 

set at, for instance, k = 1/2. 

When the direction-dependent low-pass filtering 

processing for the entire image plane is completed, the 

image processing unit 15 executes the following 
15 substitutions. 

FG [ i , j ] = Tmp_FG[i, j] 
FDR [ i , j ] = Tmp_DR [ i , j ] 
FDB [ i , j ] = Tmp_DB [ i , j ] 

20 

The electronic camera in the fourth embodiment 
achieves advantages similar to those achieved in the 
camera in the second embodiment. As a particular 
advantage to the electronic camera in the fourth 
25 embodiment, the processing scale is kept down since the 
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first similarity values are calculated by using the data 
yet to undergo interpolation processing. 

In addition, it is desirable to calculate differences 
in the characteristics of different color pixels by using 
5 color image data having undergone white balance processing. 
Since the white balance processing suppresses any extreme 
bias in color in advance, the risk of making an erroneous 
judgment with regard to similarity due to a color bias can 
be reduced. 

10 It is to be noted that the inventor of the present 

invention has confirmed that the direction-dependent low- 
pass filtering processing according to the present 
invention achieves good results regardless of whether the 
gradation conversion and the gradation correction are 

15 implemented either before or after the interpolation 

processing in the embodiment described above by conducting 
a subjective evaluation on the image quality. 

It is to be noted that while the image processing 
unit 15 executes the image processing program in the 

20 embodiment explained above, the present invention is not 
limited to this example. For instance, part of or the 
entirety of the image processing described above may be 
executed by an arithmetic circuit or the like in order to 
improve the processing speed of the image processing unit 

25 15 . 
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It is to be noted also that it is desirable to lower 
the contribution factor of the characteristics difference 
between different color pixels in a first similarity value 
when image data manifest a high degree of the saturation 
5 and to raise the contribution factor of a characteristics 
difference between different color pixels in a first 
similarity value when image data manifest a low degree of 
saturation . 

FIG. 8 is a flowchart of an operation in which an 
10 automatic adjustment of the contribution factor is 

incorporated. The following is an explanation of the 
operation added to the processing achieved in the fourth 
embodiment . 

© Step S51: calculation of two types of saturation 
15 The image processing unit 15 calculates saturations 

CCt and CCy at an ith row and a j th column in the image 
data ORG by using the following expressions. 

CCt = (|Z[i, j - 1] - Z[i, j]| + |z[i, j + 1] - Z[i, j]|) 
20 / 2 

CCy = ( |Z[i - 1, j] - Z[i, j] | + |Z[i + 1, j] - Z[i, j] | ) 

/ 2 

© Step S52 : saturation judgment 
25 The image processing unit 15 performs a judgment in 



68 




conformance to the following conditional expression. 

(CCt > BWth) and (CCy > BWth) 

5 When the two types of saturation CCt and CCy both 

exceed the threshold value BWth, it is likely that the 
current position manifests a high degree of saturation. 
In this case, the image processing unit 15 reduces the 
values of coefficients pi ~ 2 , ql ~ 2 , rl - 2 and si - 2 

10 of characteristics differences among different color 

pixels and increases the values of coefficients p3 ~ 4, q3 
~ 4, r3 - 4 , and s3 - 4 of characteristics differences 
among the same color pixels to lower the contribution 
factors of the characteristics differences among different 

15 color pixels. 

If, on the other hand, either one of the two types of 
saturation CCt and CCy is smaller than the threshold value 
BWTh, the current position is determined to manifest a low 
degree of saturation. In this case, the image processing 

20 unit 15 raises the contribution factor of the 

characteristics difference between different color pixels 
by, for instance, increasing the values of the 
coefficients pi - 2, ql ~ 2, rl - 2, and si - 2 of 
characteristics differences among different color pixels. 

25 It is to be noted that the values of the coefficients 
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may be adjusted as follows. 

(a) Complete switch between same color pixels and 
different color pixels 

If it is decided that the current position manifests 
5 a high degree of saturation, the image processing unit 15 
sets the values of the coefficients pi ~ 2 , ql ~ 2 , rl ~ 2 , 
and si - 2 for the characteristics differences among 
different color pixels to 0 and sets the values of the 
coefficients p3 - 4, q3 - 4, r3 ~ 4 , and s3 - 4 for the 

10 characteristics differences among same color pixels to 1 . 
If, on the other hand, the current position is decided to 
manifest a low degree of saturation, the image processing 
unit 15 sets values of the coefficients pi - 2, ql - 2, rl 
- 2 and si - 2 of the characteristics differences among 

15 different color pixels to 1 and sets values of the 
coefficients p3 - 4, q3 - 4, r3 -4, and s3 - 4 of 
characteristics differences among same color pixels to 0. 

(b) Using characteristics difference between the same 
color pixels alone when manifesting a high degree of 

20 saturation 

If it is decided that the current position manifests 
a high degree of saturation, the image processing unit 15 
sets values of the coefficients pi - 2 , ql ~ 2 , rl - 2 and 
si - 2 of characteristics differences among different 

25 color pixels to 0 and sets the values of the coefficients 
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p3 ~ 4, q3 ~ 4 , r3 - 4, and s3 ~ 4 of characteristics 
differences among the same color pixels to 1. If, on the 
other hand, the current position is determined to manifest 
a low degree of saturation, the image processing unit 15 
5 sets the values of the coefficients pi ~ 2, ql -2, rl - 2 
and si - 2 of the characteristics differences among 
different color pixels to 1 and also sets the values of 
the coefficients p3 - 4 , q3 - 4 , r3 ~ 4 , and s3 - 4 of 
characteristics differences among same color pixels to 1. 

10 By varying the contribution factors of the 

characteristics differences among different color pixels 
in the first similarity values in correspondence to the 
degree of saturation as described above, the risk of 
erroneously deciding that a position manifesting a high 

15 degree of saturation constitutes part of the image 

structure is reduced. As a result, it becomes possible 
to make a more accurate judgment with regard to the 
directionality of the image structure when executing the 
direction-dependent low-pass filtering processing. 

20 To be more exact, rl and si are the coefficients of 

the characteristics differences among same color pixels 
when the target position is at a G pixel, and r2 and s2 
are the coefficients of the characteristics differences 
among same color pixels when of the target position is at 

25 R and B pixels. Accordingly, the values of rl - 2 and si 
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~ 2 may be further adjusted in correspondence to whether 
the target position is at a G pixel or a R and B pixels. 
In such a case, an advantage is achieved in that an even 
more accurate judgment can be made with regard to the 
5 image structure. However, there is a risk in that 

calculated first similarity values may fluctuate depending 
upon the pixel position to result in a grid-like pattern 
manifesting after the direction-dependent low-pass 
filtering processing. If such a problem is likely to 

10 occur, it is more desirable to handle rl - 2 and si - 2 
uniformly as the coefficients of the characteristics 
differences among different color pixels as in the 
processing described earlier. 

It is to be noted that while individual methods of 

15 similarity judgment with different features are explained 
in reference to the first ~ fourth embodiments, these 
methods may be combined as appropriate or the results of 
the judgments performed the individual methods may be 
compared to use the results indicating a pronounced 

20 difference or to use the results representing the average 
of the various methods . By combining the individual 
methods in such a manner, similarity judgment achieving an 
even higher degree of accuracy is enabled. 

In addition, the names of the variables such as the 

25 first similarity values in the first ~ fourth embodiments 
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are matched with the names of variables in the claims to 
simplify the explanation. However, such a specific 
description does not necessarily limit the contents of the 
variables in the claims. 
5 It is to be noted that while an explanation is given 

in reference to the first through fourth embodiments on 
image data obtained in a matrix pixel array or a Bayer 
array, the present invention is not limited to these 
examples. For instance, the present invention may be 
10 adopted in a honeycomb pixel array or any other special 

pixel array, as long as the "at least four directions" and 
"the values of the weighting rates" are determined as 
appropriate by taking into consideration the distances 
between pixels along the individual directions. 
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